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ABSTRACT 

The purpose of this paper is to consider a range of 
qualitative aspects and measures of student learning in current use, 
and to consider the insights into student learning that these 
provide* In this study, a measure of depth of processing, the 
Structure of Observed Learning Outcomes (SOLO) taxonomy developed by 
J* B* Biggs and F* K* Collis (1982) was applied* To determine whether 
students learn in depth when using computer assisted learning (CAL) 
programs, an evaluation study was carried out on two groups of 
postsecondary level education students who used a CAL program to 
learn a particular aspect of their course* Students were askod a 
question about the material they had studied which required an open 
ended and structured response* This question was assessed using the 
SOLO taxonomy, which assessed depth of learning in terms of the 
relat ionships that student constructed, and the structured nature of 
the student's response* One study showed a high level of SOLO 
responses, which the other indicated a very low level* Research using 
the Study Process Questionnaire (SPQ) has indicated that many 
learners do not use an approach that would lead to a deep level 
response* All students were asked to complete the SPQ at the 
beginning of each study* The comparison between the SPQ and SOLO 
scores highlights the difference between the two groups of students 
evaluated* Interpretive data was gathered from both groups of 
students on their use of the program* Although neither program used 
generative learning strategies, many students in one group did show 
evidence of deep learning* (Contains 11 references*) (AEF) 
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Within the range of current investigations into the effectiveness of education and student 
learning, a wide range of measures have been developed to assess the effectiveness of 
student learning and other educational outcomes. These measures include data from case 
studies, student interviews, large scale surveys of student approaches to tertiary education, 

^ and cognitive studies of learning style and memory which, together with attainment tests, 

provide a comprehensive picture of the motivational, attitudinal and cognitive approaches 
^ taken by students, and the subsequent achievement in terms of performance. 

Q When the use of CAL and multimedia is considered, the focus of studies of student learning seems to 
be much narrower. Many of the studies of learning from CAL use a test of attainment at the end of the 
program as being the only measure of student learning. This raises questions about the nature of such 
testing, or of the actual test itself: is it primarily a test of recall or does it test the ability to reason; is 
there a variety of question styles; do the students have to apply their learning to previously 
unencountered situations; and what is the significance of the learning task to the learner? These 
questions relate to the effectiveness of the learning task in terms of its value to the student and, while 
they have wider implications than can be addressed by individual studies, they represent qualitative 
aspects that have educational significance, but are generally ignored in studies of CAL. 

The purpose of this paper is to consider a range of qualitative aspects and measures of student 
learning in current use, and to consider the insights into student learning that these provide. This will 
be considered in relation to the measures used to gauge student learning from CAL packages. An 
approach to CAL that uses qualitative measures appears to be lacking from the research literature. 

One such approach, carried out by the author, is described. In this study a measure of depth of 
processing, the Structure of Observed Learning Outcomes (SOLO) taxonomy developed by Biggs & 
Collis (1982) was applied. 




Qualitative measures of iearning 

Qualitative measures of learning are aimed at informing us about aspects of learning that do not 
emerge from conventional test scores. Aspects, such as the learning style employed by the student, the 
students’ perception of the nature and importance of the task and the approach taken, or strategy used 
to achieve learning, and the learning outcome in terms of its meaningfulness to the student or depth of 
learning are all important issues in relation to the overall educational aspect of the learning task. The 
level of student attainment is clearly related to the way in which students approach their studies. This 
approach can be considered as a style or a strategy for learning, with more effective strategies clearly 
leading to a greater level of success. An understanding of the learning strategies used by students can 
provide some guidance as to the way that learning tasks are best structured to bring about the desired 
outcome. 

To identify approaches to learning, Biggs (1987), Schmeck (1983), and Entwistle & Waterson 
(1988) have all developed questionnaires aimed at identifying approaches to learning. An example is 
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the Study Process Questionnaire developed by John Biggs (1987). In this questionnaire, students are 
asked to report on their normal approach to a wide range of aspects of the learning task. From these 
responses, it is possible to determine whether the student is intrinsically motivated to leam in depth in 
order to achieve a level of understanding, or is likely to take an approach of rote learning specific 
facts. The questionnaire also measures the extent to which achievement, in the sense of high grades, 
motivates the student. The questionnaire measures approach and strategy on each of the Deep, Surface 
and Achieving dimensions, which can be simplified to a dichotomy between Deep Achieving and 
Surface Achieving approaches. By using a questionnaire of this kind, insight can be gained into how 
particular groups of students approach their study, and also into the range of styles that are used by 
students in accomplishing a learning task. 

Questionnaires measure a basic approach, however other measures have identified that this is not 
necessarily consistent for each student across every task. Laurillard (1984) found that the depth of 
learning aimed for and achieved by students, as identified by a procedure of asking the student to 
teach back the material that had been learned plus interviewing the students about the strategy used, 
would vary according to the student’s perception of the importance of the task. This means that the 
quality of learning is influenced to a high degree by whether the student sees the task as worthy of 
deep study. Some students reported that if they thought the assessment would be a test of memory or 
recall, then an attempt to understand the material in depth was not necessary, and a more superficial 
approach would suffice. While the SPQ measures student approach, Laurillard used a qualitative 
measure of both approach and outcome. 

The identification of depth of learning as an outcome has been identified by Biggs and Collis 
(1982), who developed the SOLO taxonomy to measure this. This taxonomy is used to classify 
responses to an open-ended question, in terms of the relationships the learner draws between the 
concepts learned and whether the learner is able to structure these according to an appropriate 
conceptual framework. By . using a measure such as this, student learning can be classified according 
to the extent to which the student has created meaningful associations with the newly acquired 
knowledge, as opposed to the extent the student has memorised or reproduced the information. 

Many of these aspects are the subject of ongoing study by academics involved with research into 
the quality of teaching and learning, as they are of value for the insights they provide into what it is 
about learning tasks that motivate students to leam effectively and perform well. 

Depth of processing and CAL design 

Depth of learning is associated in cognitive terms with the concept of depth of processing first 
identified by Craik & Lockhart (1972) as being a major attribute of effective learning. This concept 
refers to the extent of the meaningful network of associations that the learner makes when learning 
new material. Craik & Lockhart identified a hierarchy of levels of cognitive processing, with greater 
depth of processing associated with the extraction of meaning from the material being learned. This 
they associated with more effective and efficient learning, as the creation and development of 
meaningful relationships make the efficient processing of large amounts of information possible. They 
characterise processing as the manipulation of conceptual schema, with efficient processing of large 
amounts of information occurring through schemas that contain a rich association of relevant 
information. 

The relationship of these aspects of the learning process to CAL development have been 
identified by Hannafin & Rieber (1989) who argue that the methods frequently used in the design of 
CAL programs, such as 

- small learning units 

- controllable sequences 

- discrete discernible steps 

- behaviourally defined objectives and performance criteria 

do not encourage deep mental processes. This implication is developed further by Jonassen (1988), 
who identifies four levels of information processing strategies; recall, integration with existing 
schema, organisation of existing and new schema, and elaboration, involving using and making 
judgements on the material. For Jonassen (1988, plS4), CAL programs require what he describes as 
“generative learning strategies”, such as “generating mnemonic memory aids, note taking, 
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underlining, paraphrasing, summarising, generating questions, creating images, outlining, and 
cognitive mapping” to induce the learner to engage in an active process of constructing meaning 
during learning. 

The argument presented by Hannafin & Rieber, and Jonassen, is that CAL programs are 
potentially superficial in the mental processes that the learner is required to engage in, and that the 
resultant learning is likely to be superficial. While this claim has intuitive merit in the sense that it is 
only by the student making an active response that learning is likely to occur or be effective, it may be 
oversimplified in that it does not take into account the mental processing that the student may engage 
in that is not required by the program, which may reflect the student’s own learning style or 
motivation. This issue is investigated by examining whether depth of processing is an outcome even if 
programs do not, as most don’t, use some form of generative learning. 

Testing for depth of processing 

The first issue is whether students do, in fact, learn in depth when using CAL programs. To do this an 
evaluation study was carried out on two tertiary level groups of students who, as part of their planned 
course of studies, used a CAL program to learn a particular aspect of their course. 

The testing procedure for depth of learning was carried out after the students had used the CAL 
program, and independently of it. The students were asked a question about the material they had 
studied which required an open ended and structured response. This question was assessed using the 
classification in the SOLO taxonomy, which assessed depth of learning in terms of the relationships 
the student constructed, and the structured nature of the student’s response. 

Each study provided a different pattern of responses, with one study showing a high level of 
SOLO responses, indicating an in-depth response by a majority of the students, and another study 
showing a very low level of student responses. In this second study, most learners reproduced 
elements of the information provided, without attempting to relate these elements in any way, either to 
other newly acquired concepts in terms of a structure, or to their own experience in a way that would 
indicate a process of assimilation with existing knowledge and cognitive structures had taken place. 
Some answers, showing a deeper, more related, response, were observed, however these were in the 
minority. These evaluations indicate a contradictory response to the issue, except to say that some 
students did respond in depth, however the studies were not directly comparable due to contextual and 
motivational factors that are considered below. 

Depth of processing in relation to learning style 

Research using the Study Process Questionnaire (SPQ) has indicated that many learners do not use an 
approach that would lead to a deep level response. The type of student approach that leads to 
memorisation of facts and procedures, rather than trying to understand the meaning and implication of 
these in relation to the learner’s own cognitive process, will most likely result in a non-relational 
response when evaluated using the SOLO taxonomy. By examining the relationship between the 
student’s tendency to learn in a deep or surface way, compared to the observed outcome of learning, 
more insight into the effectiveness of the learning program can be gained. 

All students were asked to complete the SPQ questionnaire at the beginning of each study in 
order to provide additional information on the effect of thte CAL program on learning. As the SPQ 
provides a measure of deep learning as an approach to a learning task, and the SOLO taxonomy 
provides a measure of depth of learning as an outcome of the learning task, a degree of correlation 
between the two scores can be expected. Studies by Biggs (1979) and Watkins & Hattie (1981) have 
established a correlation between the two measures, although the correlation is not necessarily strong. 
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Group A 




Group B 




SPQ SOLO 



The comparison between the SPQ and SOLO scores highlights the difference between the two 
groups of students evaluated. In Group A, in which a high level of SOLO responses was recorded, the 
comparison between the SPQ and SOLO scores shows a level of in-depth responses that is higher than 
the extent of the deep learning approach as indicated by the SPQ scores. In Group B. the level is 
lower, indicating that many students who are inclined to take a deep approach to learning did not 
respond in depth. 

These differences can only have come about through the circumstances under which the learning 
took place, including the CAL programs used. Neither program used generative learning strategies, 
however there were other differences between the programs that partly explain the differences in the 
results. There were, however, differences in the context in which the programs were used that also 
partly explain the results. In order to identify all factors that may have affected student performance, 
additional data was obtained from the students by interview and questionnaire. 

Interpretative data 

In Group A, a selection of students were interviewed about their use of the program. From this data, a 
number of trends are apparent which illuminate the data from the SPQ and SOLO scores. The trends 
in the student responses indicate; 

1 . The students found the program easy to use. 

2. The program fitted in well with the course, by providing information and learning activities 
that were important within the context of the overall learning task. 

3. While the students also used other sources of information, including a lecture and a text book, 
they mostly reported that the CAL program was the source that required them to think most 
deeply about the topic. 

4. The CAL program helped the students to form an in-depth understanding of the topic. Many 
students commented on the inclusion of a case study as being particularly helpful. 

It must be noted that the students in group A were preparing for an examination question on the 
topic, and used other sources of information apart from the CAL program. This clearly influenced the 
motivation to study the topic, to the extent that some students reported taking notes from the program 
to use later. The question that was subjected to analysis using the SOLO taxonomy was the exam 
question on the topic. 

Interpretative data was gathered from Group B also, this time by questionnaire. While the 
responses were not uniform, trends are apparent that indicate the following: 

1 . The students did not find the program easy to use. 

2. The program content was seen to be related to the course content, however the reason for using 
the program was not clear to the students. 

3. The program did not encourage the students to think deeply about the topic. 

4. The program did not enable the students to form a deep understanding of the topic. 

5. The students felt that the program did not provide enough information to effectively learn 
about the topic. 






From this information it is clear that the programs differed from each other in terms of their 
effectiveness in meeting the learning needs of the students, in part because of the context in which 
they were used, and in part due to the effectiveness of the information provided and the learning 
activities required. Contextual factors are clearly important. The students in Group A saw a clear need 
for the knowledge contained within the program, and were happy to use it. Group B students, by 
contrast, did not understand why they were asked to use the program, and did not want to use it, 
despite the fact that the content was clearly related to their course of study. This aspect of lack of 
preparation is likely to have influenced performance. 

It is difficult to assess by how much, however. The group A students were generally satisfied 
with the information provided by the program, and found it helpful in reaching an in-depth 
understanding, a self-report that is supported by the SOLO data. TTie group B students found it 
difficult to obtain the information that was needed, and reported that this did not provide a 
comprehensive explanation of the topic. These factors were clearly an important influence on the lack 
of depth that was generally evident in the results from this group. 

Conclusion 

The data from this study so far is not conclusive. Although neither program used generative learning 
strategies which, according to Jonassen (1988), means that the resulting learning will inevitably be 
superficial, many students in one group did show evidence of deep learning. As this group sat for an 
examination, it is entirely possible that the students used their own generative learning strategies, such 
as writing summaries, and that these were instrumental in attaining the in-depth learning as assessed 
by the SOLO taxonomy, even though they were not built in to the program. It is clear, however, that 
the programs differed markedly in their effectiveness as aids to learning. Further investigation is 
needed to establish the factors about program design that influence depth of learning. 
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